Enterprise Resource Planning (ERP) software systems play a crucial role in the operation of many modern businesses, and include a wide variety of complex software systems. Examples include integrated systems such as those based upon the software systems of SAP AG (Walldorf, Germany), particularly the SAP R/3 (the standard business software offered by SAP), and related systems which feature a multilevel client/server system for enterprise-wide integration. Other examples of ERP software systems include the products of PeopleSoft Inc. (Pleasanton, Calif.), such as their Enterprise Human Capital Management system; Oracle (Redwood Shores, Calif.), such as their Oracle Enterprise Manager; SSA Global (Chicago, Ill.), such as their SSA ERP (LN) system and associated software; and Siebel (San Mateo, Calif.), such as their CRM On Demand customer relationship management system and other Siebel software. Such systems may comprise hundreds or thousands of modules that are interrelated. Implementing the complex software system often requires custom programming of numerous software modules to combine software from an external vendor with the existing systems of a corporation or other entity. When a component is modified in any way during or after implementation of the system, either by the user or by vendors of the various software modules, extensive testing is needed to ensure that the integrated system performs as desired. In the past, large amounts of human labor have been expended to manually keep track of the complex relationships between components of the integrated system to determine what tests need to be repeated when changes are made. Numerous errors can be made in this process, often resulting in unnecessary testing or missed defects in the total system. Many millions of dollars are lost due to the inefficiency of prior methods for testing integrated systems.
Implementation and integration of ERP software systems and other complex software systems often involves integrating new software modules from one or more vendors with existing “legacy” systems (a corporate system that existed and functioned before bringing in the complex system from the outside vendor), and may require custom programming of interfaces to allow the new and old systems to communicate and share data. For example, a legacy system for executing systems such as sales or inventory management software may need to convey information to programs of the external vendor. This may entail preparing custom interface programs (e.g., translation software) for translating the information from the legacy system for use in the complex system. For the combined software system, with interface modules, vendor-supplied software modules, and other software modules to be successfully integrated into a complex software system, extensive integration testing is needed to ensure that the components function as intended. However, merely testing the interface modules alone may be inadequate, for testing may also be needed for any modules that use data provided through the interface module to ensure that all needed data is provided in the right form for all modules that may depend on the provided data. If the relationships between the various modules in the complex software system are not thoroughly understood, the operator doing testing may either run too many test cases, testing elements of the complex software system that have not been affected by the new or updated interface, or may fail to run the right test cases needed to verify operation of one or more affected modules.
Unfortunately, the need for extensive testing continues as individual software modules are updated. Thus, in addition to the vast amount of testing needed to validate the operations of a complex system during phase-in, any of the numerous software modules from vendors may be updated, requiring additional testing to be conducted of the updated module and those that depend on it. The process of testing to ensure an update did not change existing processing (i.e., that previously functioning system still functions as expected) is often termed “regression testing.” The process of determining what aspects of a complex software system are affected by a change in one or more software modules (or what aspects of a program are affected by a change in one or more portions of the code) is known as “impact analysis” or sometimes as “change analysis.” An exemplary work on past developments in these areas is Software Change Impact Analysis edited by Shawn A. Bohner and Robert S. Arnold, Los Alamitos, Calif.: IEEE Computer Society Press, 1996. Chapters of particular interest include Chapter 3, “Automated Support for Impact Analysis”; Chapter 4, “Dependency-Analysis Approaches”; Chapter 5, “Traceability Approaches”; and Chapter 7, “Impact-Determination Techniques.”
A variety of tools are known for automating several aspects of software testing. For example, integration and implementation of the Siebel 7 system (a customer relationship management software) is said to be assisted using tools from Empirix (Boulder, Colo.). This is described in the white paper, “implementing an Effective Web Application Testing Process” by Dan Kolaski, on Oct. 7, 2003. Principles of software testing for this system are also disclosed in the white paper, “Testing Essentials for Siebel 7 Deployments” Dan Kolaski, on Oct. 7, 2003. Generally, typical tools for automated testing include capture and playback tools for generating scripts to simulate user actions, and testing tools that can observe the actions of the software in response to automatically executed scripts.
One tool used for software testing is REQUISITEPRO® software of IBM. Other tools are marketed, for example, by Mercury Interactive (Mountain View, Calif.), such as MERCURY BUSINESS PROCESS TESTING™ software or MERCURY TEST DIRECTOR™ software; Segue Software, Inc. (Lexington, Mass.), such as their SILKTEST® software for regression and functional testing; and Empirix Inc. (Waltham, Mass.), whose E-TESTER™ software is used for providing automated functional and regression testing for Web applications, and is said to record the objects on every web page visited and automatically inserts test cases to validate the objects.
Another example of a tool for automated regression testing is described in U.S. Pat. No. 6,427,000, “Performing Automated Testing Using Automatically Generated Logs,” issued Jul. 30, 2002 to Mumford et al., herein incorporated by reference to the extent it is noncontradictory herewith. The Mumford et al. patent is said to disclose a system that performs automated testing on service applications using automatically generated logs, so that new testing applications do not need to be created for each new release of a service application. Mumford et al. further provide for testing of the service application by accounting for all data and processes performed, without interfering with the processing of the service application. An Automated Regression Tester (ART) captures and records data generated by the execution of an initial version of a service application. This data is recorded to a first test log. When a new or ported version of that service application is developed, the first test log is used by the ART to generate output that emulates the operating environment, including caller and user input, of the service application. The processing of the new/ported version of the service application is captured and recorded to a second test log. The ART then compares the second test log to the first test log and reports any discrepancies in data. Such discrepancies may indicate a bug in the new/ported version of the service application.
Another automated tool is described in U.S. Pat. No. 6,694,509, “Automated Regression Testing of Workstation Software,” issued Feb. 17, 2004 to Stoval et al., herein incorporated by reference to the extent it is noncontradictory. The Stoval patent discloses a method and means for maintaining a library of test scripts for use in regression testing application programs. More particularly, a test directory tree is maintained which is a mirror image of a source directory tree that is used to compile the executable code for the application program. The test directory tree indicates the test scripts which are to be used in regression testing executable code compiled from corresponding entries in the same directory trees. Also see U.S. Patent Publication No. US20040088602A1, “Automated Recording and Replaying of Software Regression Tests,” published May 5, 2004 by Cohen and Vollers, herein incorporated by reference to the extent it is noncontradictory herewith.
For implementation of SAP® systems, one commonly used tool is the Computer Aided Test Tool (CATT) that falls into the category of Computer Aided Software Engineering (CASE) tools. CATT is integrated into the ABAP Development Workbench of SAP® software. CATT allows those working with SAP® software to automate test flows, based on precisely defined input (batch input). After tests are run with CATT the R/3 System can automatically create a run log that is archived and serves as the basis for documentation of various test results. The test results can then be examined to evaluate the quality of the system. CATT can be used as tool to assist in the execution of the present invention. For example, it can be used to capture keystrokes and insert data to automatically execute the test case instead of having a person manually execute the transaction and fill in the data. The use of CATT for SAP® software is described in detail in Gerhard Oberniedermaier and Marcus Geiss, Testing SAP R/3 Systems: Using the Computer Aided Test Tool, transl. Jason M. Miskuly, Harlow, England: Addison-Wesley, 2000 (see particularly pages 28-213).
General principles for automated software testing are described on the Web pages of Automated Testing Specialists (Los Angeles, Calif.). Described therein is the Functional Decomposition Method, in which script development methodology is used to reduce all test cases to their most fundamental tasks. User-defined functions, business function scripts, and “sub-routine” or “utility” scripts perform these tasks independently of one another. Automated test scripts can be written for a business function, using data-files to provide both the input and the expected-results verification. A hierarchical architecture is employed, using a structured or modular design. Also described is the “Test Plan Driven” method, in which, the entire testing process is data-driven, including functionality. The test plan is prepared in a specific format that pre-written “utility” scripts use to control the processing of the automated test. Hybrid methodologies can also be used, including the ATS Toolkit for WINRUNNER® software.
Guidelines for software testing in general are discussed by C. Kaner, J. Falk, and H. Q. Nguyen in Testing Computer Software (New York: Wiley Computer Publishing, 1999). Automated regression testing is discussed on pages 191-197. Principles of automated software testing are also described in detail by Elfriede Dustin, Jeff Rashka, and John Paul, Automated Software Testing (Boston: Addison-Wesley, 1999, pages 293 and 295-306).
Impact analysis, known as the effect of changes in one software segment on other software segments, is known. Recent advances in impact analysis are described by A. Orso, T. Apiwattanapong, and M. J. Harrold, “Leveraging Field Data for Impact Analysis and Regression Testing,” ESEC/FSE'03 (Foundations of Software Engineering: Proceedings of the 9th European Software Engineering Conference held jointly with the 10th ACM SIGSOFT International Symposium on Foundations of Software Engineering), Sep. 1-5, 2003, Helsinki, Finland, printed by ACM (Association for Computing Machinery), New York, 2003, pp.128-137. Background information on impact analysis is provided by Shawn A. Bohner and Robert S. Arnold in “An Introduction to Software Change Impact Analysis,” in Software Change Impact Analysis (Los Alamitos, Calif.: IEEE Computer Society Press, 1996, pp. 1-26), herein incorporated by reference. Bohner and Robert refer to software life-cycle objects (SLOs) that include all aspects of a software project from specifications to finished programs and-they discuss the effects of making changes in the software. They note that changes in software design can be needed for many purposes, such as changes in requirements and specifications, design trade-offs, interface changes, performance problems, feedback from prototypes, etc. Changes in programs themselves can be the result of bug fixes, algorithm adjustments, data structure modifications, parameter changes, etc. In dealing with such changes, there is a need to analyze the impact of the change on other components of the system and the impact on the requirements and objectives for the software. Among known tools, Bohner and Arnold list the Dependency Analysis Tool Set developed by Norman Wilde at the University of West Florida “to assist in understanding the complex interrelationships found in large systems developed in C.” They also discuss Alicia, an impact analysis tool developed by the Air Force.
Also known in the art is the concept of a traceability matrix, which is known as a way of keeping track of the software modules, documents, etc., that are used to support objectives of the software. In the prior efforts, the traceability matrix was generally created manually, and did not contain detailed information about multiple levels of dependencies and complex interrelationships between components. Rather, known traceability matrices present what superficial, first-order (single level) view of basic relationships between software requirements or specifications and the software modules.
In general, the challenge of testing software modules to validate the operations of complex software systems such as an Enterprise Resource Planning system can easily represent one of the most difficult, costly, and man-power intensive activities within an organization. Consequently, there is a need for methods to improve the process of testing software modules and to streamline such processes. Reduction of redundant testing and elimination of unneeded test runs is desirable, but is not presently available.
In spite of the prior recognition of the need to construct a “modularity-relationship analysis,” past efforts appear to have been dominated by manual approaches to dealing with such analyses. Automated tools for regression testing that exploit a Traceability Matrix do not appear to be known, nor do there appear to be known tools that automatically generate a “roadmap” of the relationships between software modules to identify the test cases that need to be included in a regression test.
While manual or computer-based requirement tracking tools exist for relating software objectives to the modules that deliver the objectives, these tools generally do not provide detailed information on multiple levels of dependency between modules or other components of a complex software system. Indeed, while many tools have been developed to assist in the analysis of individual software programs or to automate various aspects of testing of complex software system (e.g., the CATT tool used to assist SAP implementation), there remains a need for improved automated tools for complex software systems to make testing more efficient and to solve many of the problems associated with implementing complex software systems, particularly when software from multiple vendors, including legacy components, must be integrated.
While others have proposed the use of test cases and scripts for automated software testing, sometimes in conjunction with a conventional traceability matrix, the ability to efficiently select which test cases need to be run in response to a given software modification has generally been lacking. There is a need for an improved automated software tool that allows the complex interrelationships between software modules and test cases to be identified automatically, allowing an operator to make a much more efficient determination of what fraction of the test cases need retesting. There is also a need for a tool to assist software testing that can take advantage of the data in a Traceability Matrix to provide automated roadmaps of relationships between affected test cases in order to more efficiently select test cases needed for proper testing following a change in one or more software modules or other system components.